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Summary 


Problem 

Entry  into  military  occupational  specialty  (MOS)  training  schools  requires  a 
minimum  score  requirement  on  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB),  a  test  battery  that  assesses  cognitive  abilities  through  performance  in  reading, 
mathematics,  general  science,  as  well  as  basic  knowledge  about  electronics,  mechanical 
systems,  and  automotive-shop.  The  ASVAB  was  designed  to  increase  the  effectiveness 
and  efficiency  of  the  selection  procedure,  and  determines  that  incumbents  have  the 
necessary  cognitive  skills  to  handle  the  technical  aspects  of  training  and  job 
performance.  However,  the  current  Marine  Corps  Air  Traffic  Control  selection  process 
using  the  ASVAB  has  resulted  in  cognitively  proficient  applicants,  who  are,  nevertheless, 
classified  as  “unsuited”  for  ATC  duty  during  post-training  performance.  The  inclusion  of 
individuals  into  the  ATC  MOS  who  will  ultimately  be  unfit  for  this  position  is  an 
inefficient  use  of  time  and  resources.  The  singular  focus  on  cognitive  skills  has  resulted 
in  an  unproductive  and  costly  system,  and  requires  an  updated  approach  to  screening 
and  selection. 

Objective  or  Purpose 

Individuals  possess  a  variety  of  abilities,  preferences,  interests,  and  personal 
characteristics  that  should  be  useful  to  predicting  who  will  be  best  suited  for  different 
types  of  military  jobs  or  duties.  The  objective  of  this  project  is  to  capitalize  on  these 
individual  differences  by  developing  a  non-cognitive  profile  of  successful  air  traffic 
controllers  to  be  used  in  the  selection  procedures  for  Marine  Corps  ATC  duty.  Making 
such  a  suitability  profile  will  greatly  increases  the  efficiency  of  the  Marine  Corps  ATC 
personnel  pipeline  and  reduce  the  burden  of  unproductive  training  by  providing  early 
detection  of  Marines  who  are  and  are  not  likely  to  perform  satisfactorily  as  ATCs. 

Approach  or  Method 

Using  a  multifaceted  adaptive  personality  measure,  the  Navy  Computer  Adaptive 
Personality  Scales  (NCAPS),  the  authors  assessed  ATCs  in  the  supporting  establishment 
operating  forces  (7257,  7252,  7253,  7254  MOS)  and  Marine  ATC  students  (7251  MOS) 
on  19  different  individual  traits.  Through  both  concurrent  and  predictive  validity 
designs,  scores  on  the  NCAPS  were  correlated  with  measures  of  ATC  job  and  training 
performance.  The  resulting  algorithm  was  used  to  create  a  profile  from  a  subset  of  the 
NCAPS  traits  that  demonstrates  predictive  validity  for  ATC  performance.  This  profile  of 
suitability  for  selection  as  an  ATC  was  termed  the  Marine  Air  Traffic  Controller  - 
Suitability  Test  (MATC-ST). 

Findings  or  Results 

The  subset  of  NCAPS  traits  that  demonstrated  statistically  significant  prediction  for 
ATC  (operating  forces)  performance  and  were  included  the  Marine  Air  Traffic  Controller 
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-  Suitability  Test  algorithm  were:  Adaptability/ Flexibility,  Vigilance,  Empathy,  and  Self- 
Reliance.  Hierarchical  regression  analyses  suggest  that  the  non-cognitive  suitability 
measure  adds  incremental  validity  above  and  beyond  the  ASVAB  score,  and  accounts  for 
14%  of  the  variance  of  ATC  job  performance  (R2=.i4i,  p=-375,  t=5.804,  p<.ooo).  Given 
a  MATC-ST  score,  one  can  predict  the  expected  level  of  success  an  individual  Marine  is 
likely  to  exhibit  on  the  job  and  can  make  subsequently  valid  selection  decisions. 

Conclusions 

The  Marine  ATC  Suitability  Test  (MATC-ST)  is  a  valid  algorithm  derived  from  the 
dimensions  of  NCAPS  and  has  been  shown  to  be  predictive  of  success  for  Marine  ATCs 
in  the  operating  forces.  Those  with  higher  MATC-ST  scores  advanced  and  qualified  at  a 
faster  rate  than  those  with  lower  scores.  The  expectation  is  that  the  relationship  will 
hold  up  for  future  ATC  Marines  and  that  a  near  term  goal  would  be  to  evaluate  the 
positive  effects  of  an  operational  cutscore  on  the  selection  procedures.  Further  data 
collection  and  analysis  at  the  ATC  Schoolhouse  will  provide  confirmation  of  the  MATC- 
ST’s  predictive  validity.  If  the  MATC-ST  is  to  be  utilized  for  selection/classification  it  is 
recommended  that  it  be  used  only  in  addition  to  the  ASVAB,  and  implemented  at  the 
Military  Entrance  Processing  Station  (MEPS)  prior  to  a  Marine  ATC  MOS  designation. 
This  will  reduce  attrition/revocation  and  increase  the  quality  of  Marines  selected  for  the 
ATC  MOS. 
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Introduction 


The  current  Air  Traffic  Control  selection  process  utilized  by  the  US  Marine  Corps  is 
being  reviewed  as  a  result  of  an  unacceptable  number  of  Marine  Air  Traffic  Controllers 
(ATCs)  being  classified  as  “unsuited”  for  ATC  duty  as  defined  by  performance  and 
behavioral  expectations.  The  first,  most  important  step  in  creating  a  more  efficient  force 
is  to  improve  the  Air  Traffic  Controller  (ATC)  screening/selection  process.  This 
research  effort  focused  on  developing  a  non-cognitive  Air  Traffic  Controller  profile  that 
could  be  incorporated  into  a  model  for  their  selection.  Such  a  profile  would  allow  the 
Marine  Corps  to  identify  those  Marines  who  are  likely  to  be  successful  as  Air  Traffic 
Controllers,  those  who  need  a  greater  depth  of  screening,  and  those  who  are  likely  to 
fail.  This  effort  does  not  consider  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB)  standard  that  is  established  to  minimize  ATC  academically  related  training 
failures.  Another  study  that  is  being  conducted  as  a  follow-on  to  this  research  effort  will 
address  both  the  ASVAB  standard  in  conjunction  with  the  suitability  screening  test  for 
this  purpose  and  to  replicate  this  study’s  findings. 


Background 

To  enlist  in  the  military  applicants  must  meet  the  minimum  basic  requirements  on 
the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB),  a  test  battery  that  assesses 
performance  in  reading,  mathematics,  general  science,  as  well  as  basic  knowledge  about 
electronics,  mechanical  systems,  and  automotive-shop.  Marine  Corps  applicants  who 
are  recruited  are  evaluated  on  their  ASVAB  scores  to  identify  a  list  of  military 
occupational  specialty  (MOS)  training  schools  they  are  qualified  for,  including  ATC. 
Navy  Personnel  Research,  Studies,  and  Technology  (NPRST)  has  conducted  two  studies 
(Held  &  Johns,  2002;  Held,  2006)  addressing  the  establishment  of  optimal  aptitude 
standards  for  the  Navy’s  Air  Traffic  Control  A-School.  The  recommendations  made  by 
NPRST  were  put  into  operation  for  Navy  recruits  and  have  reduced  academic  attrition, 
while  at  the  same  time  ensuring  sufficient  incoming  class  sizes. 

Though  cognitively  based  measures  such  as  the  ASVAB  help  to  ensure  that 
incumbents  have  the  necessary  information  processing  and  problem  solving  skills  to 
handle  the  technical  aspects  of  training  and  job  performance,  other  factors  also 
contribute  to  success  or  failure  in  the  occupation.  Borman,  et  al.,  (2003)  discussed  in 
their  review  of  selection  and  classification  research  that  individuals  are  more  complex 
and  multidimensional  than  can  be  measured  by  the  ASVAB  alone.  The  ASVAB  was 
designed  to  predict  cognitive  performance  -  particularly  of  performance  through 
training.  Beyond  the  cognitive  abilities  assessed  by  the  ASVAB,  however,  individuals 
possess  a  variety  of  preferences,  interests,  and  personal  characteristics  that  should  be 
useful  for  identifying  who  will  be  best  suited  for  military  missions  of  the  future. 

Many  studies  have  found  that  measuring  personality  variables  greatly  enhances  our 
ability  to  predict  who  will  perform  successfully  across  a  variety  of  jobs  in  civilian  and 
military  settings.  For  instance,  conscientiousness  is  one  of  the  best  predictors  of 
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performance  across  a  variety  of  jobs.  By  adding  a  measure  of  conscientiousness,  an 
additional  18%  of  variation  in  on-the-job  performance  can  be  explained,  over  and  above 
cognitive  ability  alone.  Another  important  personality  trait  that  accounts  for  an 
additional  10%  of  variance  in  performance  is  emotional  stability.  In  fact,  an 
investigation  with  military  participants  found  that  measuring  emotional  stability 
accounted  for  an  additional  38%  increase  in  incremental  validity  (see  Ferstl,  Schneider, 
Hedge,  Houston,  Borman,  &  Farmer,  2003).  The  established  literature  further 
demonstrates  that  personality  traits  can  predict  training  performance. 

NPRST  has  developed  and  utilizes  the  Navy  Computer  Adaptive  Personality  Scales 
(NCAPS),  a  multifaceted  personality/ character  instrument  for  selection  and 
classification.  Items  are  presented  in  a  paired-comparison  format,  and  responders  are 
required  to  choose  the  item  in  the  pair  that  is  most  closely  representative  of  them.  This 
forced  choice  format  has  been  shown  to  be  more  resistant  to  faking  than  other  forms  of 
response  distortion  (Jackson,  Wroblewski,  &  Ashton,  2000). 

NCAPS  is  an  adaptive  instrument,  where  item  selection  occurs  through  an  algorithm 
that  incrementally  increases  or  decreases  item  difficulty  according  to  the  examinees’ 
true  ability.  That  is,  if  a  participant  responds  correctly  to  an  item,  then  he  or  she  is 
presented  with  a  more  difficult  item.  Items  are  presented  in  this  fashion  until  the 
participant  consistently  answers  items  correctly  at  a  specific  level  of  difficulty  with  the 
lowest  possible  (or  acceptable)  level  of  measurement  error  (Bartram,  1993). 

In  personality  testing  there  is  no  correct  or  incorrect  answer  and  “difficulty”  does  not 
take  on  the  standard  meaning  as  in  an  ability  test.  Instead  a  “difficult”  item  is  one  that 
is  higher  on  the  trait  of  interest  (e.g.,  on  a  measure  of  extraversion,  “I  like  parties”  would 
be  considered  a  higher  level  of  a  trait  whereas  “I  like  libraries”  would  be  considered  a 
lower  level  of  a  trait).  As  the  respondent  proceeds  through  the  test,  choosing  an  item 
that  is  higher  on  the  trait  being  measured  will  cause  the  next  pair  of  items  to  be 
representative  of  slightly  higher  levels  of  that  trait.  Items  are  presented  in  this  fashion 
until  the  participant  consistently  endorses  items  at  a  specific  level  of  the  trait. 

The  purpose  of  computer-adaptive  testing  is  to  present  items  that  are  informative 
about  the  test  taker  and  to  maximize  the  precision  of  measurement  in  a  limited  amount 
of  testing  time.  For  example,  on  the  traditional  paper  and  pencil  version  of  a  cognitive 
ability  test,  a  high  ability  person  will  receive  the  same  easy  items  as  everyone  else,  yet 
the  easy  items  will  contribute  little  to  no  information  about  the  person’s  actual  ability. 

By  using  an  adaptive  format,  the  high  ability  person  will  not  be  administered  the  easy 
items,  as  it  is  expected  they  are  likely  to  be  able  to  answer  these  correctly.  By  only 
administering  items  that  are  informative  of  the  person’s  ability,  the  number  of  test  items 
can  be  greatly  reduced  along  with  the  administration  time  (Wainer  &  Mislevy,  2000).  A 
similar  approach  is  taken  when  measuring  a  person’s  personality  trait  level  using 
NCAPS.  An  individual  who  consistently  endorses  items  high  on  a  particular  trait 
domain  will  not  be  presented  with  items  on  the  very  low  end  of  that  trait  scale;  as  they 
are  likely  to  simply  choose  the  other  option,  without  adding  information  to  their  overall 
score  on  the  domain. 

Solely  owned  by  the  Department  of  the  Navy,  NCAPS  gained  recognition  in  2010 
from  the  Society  for  Industrial  and  Organizational  Psychology,  for  technological 
innovation  of  applied  psychology  in  the  workplace.  NCAPS  has  been  found  to  be  a  valid 
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predictor  of  on-the-job  performance  and  is  highly  resistant  to  faking  (Houston  et.  al., 
2006;  Underhill  et.  al.,  2008).  NPRST  is  currently  conducting  NCAPS  validation  studies 
for  many  Navy  jobs  and  developing  tailored  selection  systems.  For  example,  a  subset  of 
NCAPS  is  part  of  a  comprehensive  selection  system  for  the  Navy  Special  Operations 
(SEAL)  community.  NCAPS  selection  system  development  work  is  also  currently 
ongoing  for  other  Navy  special  operations  groups  (i.e.,  EOD,  Navy  Divers,  SWCC,  and 
AIRR)  as  well  as  with  Navy  recruiters. 

This  research  with  the  Marine  Corps  ATC  community  expanded  opportunities  to 
develop  a  comprehensive  selection  system  for  Marine  ATCs  that  includes  a  tailored 
NCAPS  that  goes  beyond  predicting  just  training  outcomes.  As  a  starting  point,  Dibble 
(2007)  identified  ATC  relevant  non-cognitive  traits  from  subject  matter  experts.  These 
traits  include  professionalism,  working  cooperatively,  personal  tolerance,  work/effort, 
self-esteem,  aggressiveness,  attention  to  detail,  task  closure/thoroughness,  decisiveness, 
consistency,  flexibility,  concentration,  composure,  tolerance  for  high  intensity  work 
situations,  and  commitment  to  the  job.  Many  of  these  traits  are  embodied  in  the  NCAPS 
and  a  composite  of  NCAPS  facets  recently  developed  to  measure  a  constellation  of 
vigilance  and  diligence  has  demonstrated  validity  for  predicting  job  performance  for  a 
number  of  Navy  ratings.  Therefore,  NPRST  is  uniquely  qualified  to  execute  research 
necessary  to  build  an  Air  Traffic  Controller  selection  system  for  the  Marine  Corps. 

This  research  project  addressed  the  studies  and  technology  element: 

•  What  individual  characteristics  will  predict  success  in  the  Marine  Corps  ATC 
military  occupational  specialty  (MOS)  operating  forces  assignments? 


Method 

In  personnel  selection  and  classification  it  is  customary  to  develop  measures  that 
predict  job  performance  and/or  job  tenure.  Measures  given  to  job  applicants  need  to 
assess  the  knowledge,  skills,  and  abilities  necessary  for  successful  performance  in  a 
particular  job,  ideally  without  producing  adverse  impact  (large  mean  differences)  for 
racial,  ethnic,  or  gender  groups.  This  study  relied  on  a  mix  of  predictive  and  concurrent 
validity  designs  using  NCAPS  and  Marine  ATC  performance  measures.  A  concurrent 
validity  design  is  when  the  predictor  is  administered  to  participants  close  in  time  to 
when  the  criterion  (performance  measure)  is  collected  (e.g.,  on  the  job).  NCAPS  was 
administered  in  such  a  concurrent  validity  design.  A  predictive  validity  design  is  when 
the  predictor  is  administered  long  in  advance  to  all  applicants  who  have  yet  to  be 
selected  for  the  job,  and  thus  represents  a  more  realistic  setting  in  which  the  instrument 
will  ultimately  be  used.  The  ASVAB,  to  be  compared  with  NCAPS,  was  administered  in  a 
predictive  validity  design. 

The  validation  approach  chosen  for  this  project  was  a  criterion-related  strategy.  This 
is  traditionally  accomplished  by  obtaining  the  test  scores  of  job  applicants  as  predictors 
and  then  collecting  measures  of  these  same  individuals’ job  performance,  the  criterion 
(or  criteria  if  more  than  one  type  of  measure  is  collected).  The  predictor  test  scores  are 
then  related  statistically  to  how  well  individuals  perform  on  the  job  and,  if  successful, 
can  be  used  to  identify  the  most  qualified  candidates  for  the  position  by  predicting  how 
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individuals  with  particular  test  scores  will  likely  perform.  This  validation  methodology 
is  one  of  three  validation  strategies  presented  in  the  Uniform  Guidelines  on  Employee 
Selection  Procedures  (1978,  EEOC),  the  Standards  for  Educational  and  Psychological 
Testing,  and  the  Society  for  Industrial/  Organizational  Psychology’s  (SIOP)  Principles 
for  the  Validation  and  Use  of  Personnel  Selection  Procedures  (1987). 

In  order  to  statistically  perform  this  validation,  measurements  of  job  performance 
and  NCAPS  scores  were  provided  for  Marines  currently  training  at  all  stages  of  the 
pipeline  (both  concurrent  and  predictive  data).  This  included  Marine  ATCs  in  the 
operating  forces  as  well  as  any  new  Marine  ATC  arrivals  at  the  ATC  Schoolhouse. 

Data  Collection 

Data  were  collected  from  Marine  ATC  students  (7251 MOS)  at  the  Air  Traffic 
Controller  Schoolhouse  in  Pensacola,  Florida  and  from  Marine  ATCs  (7257,7252,7254 
MOS)  at  eight  different  Marine  Corps  Air  Station  Air  Traffic  Control  Facilities  (MCAS 
ATCFs).  Data  collection  consisted  of  interviews,  observations,  performance 
measurements,  and  NCAPS  scores. 

ATC  Schoolhouse 

The  14-week  school  consists  of  three  blocks  of  training  that  have  knowledge  based 
written  tests  and  performance  based  assessments.  Block  One  training  is  six  weeks  long, 
entirely  knowledge  based,  and  concludes  with  the  FAA  qualification  written  exam.  Block 
Two  consists  of  local  control,  ground  control,  and  flight  data  performance  based 
assessments.  Block  Three  consists  of  basic  radar,  ASR,  PAR,  and  Arrival  written  tests 
and  performance  based  assessments. 

The  schoolhouse  requires  a  70%  or  higher  grade  to  pass,  however  in  the  operating 
forces  an  80%  or  higher  is  required  to  receive  qualifications.  ATC  officials  view  ATC 
school  as  preliminary  and  expect  that  more  extensive  learning  occurs  with  On-the- Job- 
Training  (OJT),  the  platform  MOS,  and  other  required  operating  force  qualifications  to 
maintain  qualified  controller  status. 

Schoolhouse  Instructor  /Staff  Interviews 

NPRST  researchers  met  with  school  officials  to  discuss  the  study  objectives  and 
methods,  and  the  aptitude  strengths  and  weaknesses  of  incoming  trainees.  Researchers 
were  given  tours  of  the  classrooms;  the  tower  simulator  and  other  computer  based 
classroom  training,  and  allowed  individual  time  to  interview  key  instructors  in  this 
process.  Arrangements  were  also  made  for  collection  of  performance  measures  for 
students. 

•  Marine  Corps  uses  an  ASVAB  GT  (VE,  Verbal  +  AR,  Arithmetic  Reasoning  +  MC, 
Mechanical  Comprehension)  score  of  110  as  a  minimum  qualifier.  A  Marine 
recruit  is  then  designated  into  Aviation  and  disseminated  between  ATC,  Weather, 
and  UAV.  Instructors  also  confirmed  that  this  process  leads  to  motivational 
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concerns  as  to  whether  Marines  selected  for  ATC  School  have  the  motivation  to 
be  there. 

•  According  to  instructor/Staff  interviews,  trends  were  reported  in  higher  Block 
One  student  failures  of  the:  General  Tower,  ATC  Terminal  Procedures,  and 
Emergency/Non-Radar  Exams;  higher  Block  Two  student  failures  of  the  Local 
Control  Exam;  and  no  trends  reported  for  Block  Three  student  failures. 

•  For  performance  based  evolutions  (Blocks  Two  and  Three)  students  are  evaluated 
with  critique  sheets  that  are  filled  out  for  each  evolution  that  include  instructor 
comments.  The  Critique/Evaluation  sheets  are  based  on  the  position  being 
evaluated  (Local  Control,  Arrival,  etc.)  and  students  are  given  three  scores:  (S) 
satisfactory,  (N)  needs  improvement,  (U)  unsatisfactory.  Instructors  give 
recommendations  for  improvement  on  these  critique  sheets.  A  student’s 
performance  folder  is  collected  upon  graduation/drop  and  destroyed  after  two 
years. 

Other  comments  concerning  motivation  for  Marine  7251s  from  instructors  and  staff 
were: 

•  The  instructors  commented  on  attributes  besides  intelligence  that  make 
successful  7251s.  Some  mentioned  skills  like  multitasking,  such  as  “being  a 
bartender”,  is  comparable  to  being  an  ATC.  Others  commented  that  skills  like 
achievement  motivation  and  perseverance  count  the  most  in  getting  through  the 
ATC  School  successfully. 

•  “Students  don’t  fear  authority.” 

•  “They  don’t  see  a  problem  with  quitting.  If  I  put  a  bell  on  the  quarterdeck,  they 
would  ring  it  here!” 

•  “They  don’t  see  a  problem  with  failing.  They  treat  it  like  a  video  game.  They  make 
a  mistake  and  they  want  to  push  reset  and  start  over.” 

•  Remediation  and  re-test  is  guaranteed  to  students  who  have  failed  an  exam. 
Instructors  noted  that  those  students  that  reach  out  for  help,  use  labs  outside  of 
class  time  and  that  ask  questions,  or  take  initiatives  to  approach  instructors  after 
class  are  the  most  successful  at  successfully  completing  ATC  School. 

Schoolhouse  Per  formance  Measures 

In  order  to  measure  performance  at  the  ATC  schoolhouse  we  collected  Grade  Point 
Averages  (GPA)  for  each  Marine  student  at  each  block  of  training.  In  the  context  of  this 
report,  GPA  is  a  composite  score  consisting  of  the  mean  average  for  each  block  of 
training.  GPAs  were  collected  at  the  end  of  Block  1,  Block  2,  and  Block  3  for  each  Marine 
ATC  student  (7251). 

NCAPS  at  Schoolhouse 


The  Navy  Computer  Adaptive  Personality  Scales  (NCAPS)  is  a  web-based  non- 
cognitive  measurement  tool  consisting  of  19  dimensions.  NCAPS  was  administered  to 
incoming  and  current  7251s. 
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MCAS  ATC  Facilities 


After  reviewing  the  existing  job  analysis  information,  sites  visits  were  conducted  to 
observe  controllers  from  the  various  MCASs.  The  primary  purpose  of  these  initial  site 
visits  was  to  gain  a  better  understanding  of  the  ATC  job,  the  on-the-job  training  at  each 
of  the  MCASs,  and  to  speak  with  SMEs  and  supervisors  about  the  training  process. 
Observations  were  made  of  the  controllers  from  various  radar  and  tower  positions 
performing  their  job,  and  researchers  discussed  the  various  components  of  the  job  and 
training  process  with  the  controllers,  their  trainers,  and  supervisors.  In  some  cases, 
supervisors  identified  high  and  low  performers  within  their  teams  and  described 
qualities  or  abilities  necessary  for  high  performing  ATC  Marines. 

When  a  7251  arrives  to  a  MCAS  ATC  Facility  (ATCF)  they  endure  an  indoctrination 
process  that  consists  of  an  assignment  to  an  On-the-job  Instructor  (OJTI),  testing,  and 
orientation  syllabus.  Each  MCAS  ATCF  has  an  indoctrination  process  to  help  a  7251 
understand  the  process  to  7257  qualification  at  that  particular  ATCF.  Each  MCAS  ATCF 
also  has  permanent  qualified  civilian  ATCs  as  training  specialists  to  ensure  continuity. 

Instructor /Staff  Interviews  by  MCAS  ATCF 

Site  visits,  observations  and  interviews  were  conducted  with  seven  out  of  nine 
MCAS  ATCFs.  However,  due  to  proximity  and  time  restrictions  instructor/staff 
interviews  and  site  visits  were  not  conducted  at:  MCAS  Iwakuni,  Japan,  and  MCAS 
Futenma,  Japan. 

MCAS  Beaufort,  South  Carolina 


MCAS  Beaufort  ATCF  handles  military  and  civilian  air  traffic.  They  oversee  0-10,000 
ft  within  their  jurisdiction.  Jacksonville  Center  handles  flights  11k  and  above.  Other 
major  airfields  include:  Ridg eland  to  the  West,  Walterboro  to  the  North,  and  Hilton 
Head  to  the  South.  Hilton  Head  airport  is  the  busiest  and  handles  upwards  of  25,000 
flights  per  year. 

MCAS  Beaufort  ATCF  has  13  civilian  controllers  that  remain  at  the  Facility  for 
training  continuity.  The  military  members  usually  rotate  every  three  years.  This  is 
standard  practice  for  Marine  ATCFs.  MCAS  Beaufort  ATCF  trains  and  qualifies  7257s, 
and  7253/7254S  on  a  24  hour  three  crew  rotation.  The  training  specialist  has 
implemented  a  syllabus  for  training  that  has  allowed  a  structured  format  for  learning 
and  adequate  practice  hours  for  each  ATC  Marine.  According  to  the  training  specialist  at 
MCAS  Beaufort  ATCF  this  formalized  syllabus  approach  to  training  has  increased 
productivity  for  Beaufort  ATC  and  decreased  time  (months)  to  qualification.  They 
regularly  train  ATC  Marines  on  the  AT  Coach  embedded  system  by  Raytheon.  They  also 
utilize  the  Tower  Simulator  and  recommend  it  for  ATC  Marines  with  less  experience 
because  it  gives  them  significant  practice  hours  with  an  Instructor. 

MCAS  Beaufort  ATCF  SMEs  commented  that  trends  in  7251  knowledge  based 
deficiencies  are  in:  phraseology,  separation  standards,  and  weather.  SMEs  for  Beaufort 
ATCF  also  commented  that  the  aeromedical  screening  needs  to  be  more  precise  for  7251 
selection,  for  instance  vision  issues,  back  problems,  medications  that  affect  alertness, 
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and  mental  depression,  which  can  all  lead  to  an  ATC  Marine  being  designated  not 
physically  qualified  (or  not  aeronautically  adaptable).  Some  other  factors  that  Beaufort 
ATCF  agreed  on  as  successful  attributes  in  ATCs  performance  are:  adaptability, 
motivation  to  train  and  to  learn,  a  3D  (third  dimension)  mentality,  and 
abstract  thinking  ability. 

MCAS  Cherry  Point,  North  Carolina 

MCAS  Cherry  Point  ATCF  handles  military  and  civilian  air  traffic.  They  oversee 
airspace,  0-10,000  ft  within  their  half  of  their  jurisdiction  and  0-18,000  ft  in  the  other 
half.  There  are  six  restricted  areas  and  one  other  Marine  Corps  Air  Station  (New  River) 
within  their  airspace.  Surrounding  airways  include:  Coastal  Carolina  Regional  Airport  in 
New  Bern  (due  north)  and  servicing  Delta  and  US  Airways  as  well  as  civil  traffic;  and 
Michael  J.  Smith  Field  in  Morehead  City  (due  south)  which  is  an  uncontrolled  airport 
serving  civil  aircraft. 

MCAS  Cherry  Point  ATCF  has  20  civilian  controllers  that  remain  at  the  Facility  for 
training  continuity.  The  military  members  usually  rotate  every  three  years.  This  is 
standard  practice  for  Marine  ATCFs.  MCAS  Cherry  Point  ATCF  trains  and  qualifies 
7257s,  7252s  and  7253/7254S  on  a  24  hour  three  crew  rotation. 

The  MCAS  Cherry  Point  ATCF  training  specialist  works  closely  with  the  OJTIs  to 
match  incoming  725 l’s  with  their  crews.  There  is  an  indoctrination  process  at  MCAS 
Cherry  Point  ATCF  but  they  lean  towards  a  “college  approach”  or  a  “more  flexible 
testing”  process  when  it  comes  to  written  exams.  New  7251s  are  provided  the  materials 
but  written  exam  scores  are  not  kept  on  file.  Skill  Checks/Performance  evaluations  are 
maintained  for  25%,  50%,  75%  and  100%  of  training.  The  radar  control  room  has  the 
capability  to  use  a  Raytheon  Star,  aka  the  AT  Coach.  The  Tower  Simulator  was  available 
for  required  training  hours,  but  was  considered  to  be  “inconvenient”  and  “unnecessary” 
given  the  amount  of  live  air  traffic  routed  through  Cherry  Point.  According  to  the 
training  specialist  at  MCAS  Cherry  Point  ATCF  average  time  to  quality  7257s  at  MCAS 
Cherry  Point  is  four  months. 

MCAS  Cherry  Point  ATCF  SMEs  agreed  that  there  are  some  trends  in  ATC 
performance: 

“Some  7251’s  are  really  smart,  but  can’t  apply  it  (knowledge);  They  lack  common 
sense  or  they  can’t  improvise.” 

“They  can’t  multitask  or  have  problems  thinking  3- dimensionally.” 

“Sometimes  lateral  movers  qualify  faster.  It  may  be  because  of  more  seasoning 
(experience)  and  self-confidence.” 

MCAS  New  River,  North  Carolina 

MCAS  New  River  ATCF  handles  military  and  civilian  air  traffic.  They  are  responsible 
for  Class  D  (surface  to  and  including  2500ft  with  5nm  radius  around  NCA)  and  Class  E 
(700  to  3000  ft  with  a  7NM  radius  around  NCA)  airspace.  Surrounding  airports  include: 
Albert  Ellis  (NW),  MCALF  Bogue  Field  (NE),  Wilmington  (SW),  Kinston  (N),  and 
Cherry  Point  (NE). 
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MCAS  New  River  has  ten  civilian  controllers  that  remain  at  the  Facility  for  training 
continuity.  The  military  members  usually  rotate  every  three  years.  This  is  standard 
practice  for  Marine  ATCFs.  MCAS  New  River  ATCF  trains  and  qualifies  7257s,  7252s 
and  7253s  utilizing  a  two  crew  rotation. 

The  MCAS  New  River  ATCF  Training  and  Standardization  Officer  works  closely  with 
the  Non-Commissioned  Officer-in-Charge  when  assigning  7251s  to  a  crew.  Their 
indoctrination  process  includes  a  syllabus  with  five  knowledge  based  exams.  During  this 
visit  the  Tower  Simulator  was  in  need  of  repair.  MCAS  New  River  relies  on  a  rigid  exam 
process  that  requires  memorization  of  phraseology  and  four  tests  per  ATC  position. 
Training  consists  of  daily  evaluations,  simulations  and  skill  checks. 

MCAS  New  River  ATCF  SMEs  agreed  that  there  are  some  trends  in  ATC 
performance: 

“There  is  a  lack  of  desire  and  motivation  (in  some  7251s).  They  don’t  have 
that  drive  to  do  the  work  or  the  motivation  to  prepare  for  continuous 
evaluations.. It  can  be  a  safety  issue,  when  they  (7251s)  lack  the  motivation  to  be  here 
or  express  that  they  ‘didn’t  want  this  job’.” 

Sometimes  a  lack  of  motivation  is  expressed  with  “an  I  don’t  care  if  I  fail  (as  an 
ATC),  because  I  will  still  get  promoted  and  I  still  get  paid  (as  a  Marine)’  attitude.” 

“It  is  important  to  be  able  to  think  on  your  feet.  They  need  to  be  able  to 
communicate.  Introverts  may  have  a  hard  time  with  this  job.” 

MCAS  Quantico ,  Virginia 

MCAS  Quantico  ATCF  handles  military  and  civilian  air  traffic.  They  are  responsible 
for  03000ft.  Surrounding  airports  include:  Stafford  (SW),  Andrews,  Dulles,  Reagan 
(N),  and  Richmond  (S).  There  are  also  responsible  for  a  Special  Flight  Restriction  Area 
(SFRA)  over  the  Capital  that  requires  unique  clearing  procedures. 

MCAS  Quantico  has  three  civilian  controllers  that  remain  at  the  Facility  for  training 
continuity.  The  military  members  usually  rotate  every  three  years.  This  is  standard 
practice  for  Marine  ATCFs.  MCAS  Quantico  trains  and  qualifies  7257s,  7252s  and  7253s 
utilizing  a  two  crew  rotation. 

The  MCAS  Quantico  ATCF  Training  Chief  works  closely  with  the  OJTIs  when 
assigning  7251s  to  a  crew.  Their  indoctrination  process  includes  a  syllabus  with 
knowledge  based  exams.  They  rely  heavily  on  the  Tower  Simulator  due  to  very  low  air 
traffic  in/out  of  Quantico.  Training  also  consists  of  daily  evaluations,  simulations  and 
skill  checks. 

MCAS  Quantico  ATCF  SMEs  agreed  that  there  are  some  trends  in  ATC  performance: 

“7251s  without  prior  experience  can  be  ‘lost  in  the  sauce’  and  lack  maturity  and 
common  sense.” 

Staff  recommended  that  7251s  should  not  be  sent  to  Quantico  due  to  a  low 
volume  of  air  traffic.  “A  weak  Quantico  controller  is  a  revocation  somewhere  else.” 
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“They  need  some  natural  abilities,  like  aggressiveness.  They  need  to  be 
assertive  on  position.  Shy,  quiet  people  will  struggle  in  this  job.” 

“Confident  controllers  know  that  they  can  do  the  job.  If  a  controller  is  unsure  of 
themselves  then  they  can’t  do  the  job.” 

MCAS  Yuma,  Arizona 

MCAS  Yuma  ATCF's  airspace  is  rectangular  and  approximately  90  miles  by  70  miles 
from  surface  -  23,000  ft.  There  are  multiple  restricted  areas,  and  Special  Use  Airspace 
(Ranges),  which  are  included  within  the  airspace.  Yuma  ATCF  handles  every  type, 
model,  and  series  of  aircraft  (commercial,  private,  and  military)  and  unmanned  aerial 
systems  as  well.  Adjacent  airports  include:  NAF  El  Centro  (W),  Imperial  (W),  Luke  AFB 
(NE),  and  Laguna  AAF  (N).  Yuma  ATCF  is  the  only  Joint  (Military/Civilian)  MCAS,  and 
also  the  DoD's  only  Enroute  RADAR  Facility,  as  such  they  coordinate  with  both  Los 
Angeles  Center  and  Albuquerque  Center. 

MCAS  Yuma  has  28  civilian  controllers  that  remain  at  the  Facility  for  training 
continuity.  The  military  members  usually  rotate  every  three  years.  This  is  standard 
practice  for  Marine  ATCFs.  MCAS  Yuma  ATCF  trains  and  qualifies  7257s,  7252s  and 
7253/7254S,  operating  for  24  hours  on  a  three  crew  rotation. 

The  MCAS  Yuma  ATCF  training  specialist  works  closely  with  the  Training  Chief 
when  assigning  7251s  to  a  crew.  Their  indoctrination  process  includes  orientation  and 
academic  training  with  their  Primary  Position  Instructor  (PPI).  Training  consists  of 
daily  simulations,  familiarization  time,  and  skill  checks. 

MCAS  Yuma  ATCF  SMEs  agreed  that  there  are  some  trends  in  ATC  performance: 

“There  is  a  different  mindset  (from  7251s)  out  of  the  schoolhouse.  It  used  to  be  a 
privilege  to  make  it  out  (of  school)  because  a  lot  of  Marines  didn’t  make  it.  We  actually 
feared  failure.  Now  there  is  no  sense  of  competition  or  incentives  to  be  the  best.” 

“The  successful  7251s  I  see  are  quick  thinkers-  they  can  think  on  their  feet. 
If  I  ask  them  a  question  and  they  hesitate  or  can’t  answer  then  that’s  not  good.  Also 
good  (air  traffic)  controllers  have  confidence  and  are  sure  of  themselves.  If  they 
are  too  quiet,  talk  low,  or  afraid  to  voice  their  opinion  then  they  will 
probably  not  make  it.” 

“Over  thinkers  have  a  hard  time  in  this  field-  ‘paralysis  by  over  analysis’.” 

“The  best  controllers  have  awareness  and  multitasking  ability.  They  take 
some  personal  initiative  to  study  at  home  or  seek  out  their  PPI  instead  of  waiting  to 
be  told  what  to  do  next.” 

MCAS  Camp  Pendleton ,  California 

MCAS  Camp  Pendleton  ATCF  primarily  handles  military  air  traffic  in  and  out  of 
MCAS  Pendleton.  They  also  control  instrument  arrivals/departures  into  and  out  of  one 
satellite  airport,  Fallb rook  Airpark,  which  is  located  approximately  7NM  Northeast  of 
MCAS  Camp  Pendleton.  They  oversee  a  17  by  7  mile  area  that  contains  two  sections: 
Area  A  and  Area  B.  Area  A  is  a  small  area  overlying  the  Camp  Pendleton  airport  and 
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extends  to  the  southwest  approximately  five  miles.  They  are  responsible  for  surface  to 
3000  ft  in  this  area.  Area  B  is  the  larger  area  that  extends  northeast  toward  the 
Temecula  Valley.  They  are  responsible  for  surface  to  5000  feet  in  this  area. 

MCAS  Camp  Pendleton  has  nine  civilian  controllers  that  remain  at  the  facility  for 
training  continuity.  The  military  members  usually  rotate  every  three  years.  This  is 
standard  practice  for  Marine  ATCFs.  MCAS  Camp  Pendleton  ATCF  trains  and  qualifies 
7257s,  7252s  and  7253s,  utilizing  a  two  crew  rotation. 

The  MCAS  Camp  Pendleton  ATCF  Training  and  Standardization  Officer  works 
closely  with  the  Tower  and  Radar  Training  Chiefs,  and  Crew  Officers/Chiefs  when 
assigning  7251s  to  a  crew.  Their  indoctrination  process  includes  orientation  and 
academic  training  with  their  OJTIs.  After  assigned  to  a  crew,  they  utilize  the  AT  Coach, 
IPART,  PAR  and  Tower  Simulator.  Training  consists  of  daily  simulations,  evaluations, 
and  skill  checks. 

MCAS  Camp  Pendleton  ATCF  SMEs  agreed  that  there  are  some  trends  in  ATC 
performance: 

“Lateral  movers  do  very  well.  Their  maturity  levels,  Marine  leadership  and 
experience  is  an  asset.” 

“Great  controllers  use  personal  initiative  and  have  a  natural 
aggressiveness.  ” 

“Sometimes  the  new  7251s  are  too  timid  and  hesitant  to  talk  to  others.  That 
shows  and  it  hurts  them  (their  performance)  if  they  are  hesitant  to  talk  to  aircraft.” 

MCAS  Miramar ,  California 

MCAS  Miramar  ATCF  primarily  handles  military  air  traffic.  They  are  located  in  San 
Diego  just  north  of  Montgomery  Field.  They  provide  ATC  Tower  and  GCA  services 
North  of  Highway  52,  within  Class  B  airspace  from  4  miles  to  the  Northeast,  surface  to 
and  including  3000  ft. 

MCAS  Miramar  has  eight  civilian  controllers  that  remain  at  the  Facility  for  training 
continuity.  The  military  members  usually  rotate  every  three  years.  This  is  standard 
practice  for  Marine  ATCFs.  MCAS  Miramar  ATCF  trains  and  qualifies  7257s,  and  7252s 
utilizing  a  two  crew  rotation.  The  MCAS  Miramar  ATCF  Branch  and  Training  Chiefs 
work  closely  with  crew  leadership  personnel  and  OJTIs  when  assigning  7251s  to  a  crew. 
Their  indoctrination  process  includes  orientation  and  academic  testing. 

MCAS  Miramar  ATCF  SMEs  agreed  that  there  are  some  trends  in  ATC  performance: 

“To  be  successful  you  need  an  ‘A-type’ personality .  Don’t  settle  for  second  best 
and  always  wanting  to  do  better.  This  type  of  attitude  puts  7251s  farther  ahead.” 

“They  have  to  have  good  study  habits  and  time  management.  They  have  to 
retain  a  lot  ofinformation  but  also  be  able  to  apply  it.” 
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Operating  Forces  Performance  Measures 

Performance  measures  were  collected  from  nine  MCAS  ATCFs.  The  performance 
measures  consisted  of: 

•  GT  Scores  (VE/AR/MC) 

•  Start  Date  of  7257  training 

•  7257  Qualification  date 

•  7257  written  exam  scores/GPA 

•  Start  Date  of  7252  training 

•  7252  Qualification  date 

•  7252  written  exam  scores/GPA 

•  Start  Date  of  7253/54  training 

•  7253/54  Qualification  date 

•  7253/54  written  exam  scores/GPA 

NCAPS  in  Operating  Forces 

NCAPS  was  administered  to  7251s,  7257s,  7252s,  and  7253/54S,  currently  in  the 
operating  forces  at  nine  different  MCAS  ATCFs.  The  nine  MCAS  ATCFs  were:  MCAS 
Beaufort,  SC;  MCAS  Cherry  Point,  NC;  MCAS  New  River,  NC;  MCAS  Quantico,  VA; 
MCAS  Yuma,  AZ;  MCAS  Camp  Pendleton,  CA;  MCAS  Miramar,  CA;  MCAS  Futemna, 
and  MCAS  Iwakuni,  Japan. 

Validity  Analysis 

Validity  refers  to  the  correlation  between  a  predictor  of  success  (such  as  the  ASVAB 
GT  Score,  or  NCAPS)  and  an  objective  performance  measure  (such  as  school  grades  or 
graduation  status,  or  time  to  7257  qualification).  The  ASVAB  (GT  Score)  has 
demonstrated  validity  for  predicting  job  performance,  but  is  most  predictive  of 
academic  performance.  NCAPS  has  demonstrated  validity  for  predicting  job 
performance.  The  correlation  between  the  two  predictors  is  dependent  upon  the 
constructs  being  measured  by  the  specific  criterion. 

Performance  Measures 

Job  performance  is  a  multifaceted  construct  that  can  be  measured  in  a  variety  of 
ways  (peer  ratings,  supervisor  ratings,  customer  ratings,  output,  qualifications,  absence 
of  errors,  etc.).  For  the  present  study,  a  variety  of  proxies  for  job  performance  were 
considered,  but  the  following  provided  the  most  significant  results. 
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Schoolhouse  performance  measures 


GPA  scores  for  Block  One,  Block  Two,  and  Block  Three  were  collected  and 
standardized.  ASVAB  (GT  Scores)  were  also  collected. 

Operating  Forces  performance  measures 

While  some  Marines  are  able  to  master  the  position  and  pass  the  qualification  exams 
within  a  few  months,  others  take  much  longer.  Marines  who  are  able  to  meet 
qualification  at  a  quicker  pace  are  often  considered  better  performers  and  more 
proficient  controllers.  Therefore,  time  to  quality  may  be  considered  a  proxy  for  job 
performance,  in  that  these  individuals  are  more  naturally  suited  to  the  demands  of  ATC 
positions,  and  thus  require  less  time  to  exhibit  proficient  performance. 

Time  to  quality  was  calculated  from  training  start  date  to  qualification  date  for  7257, 
7252,  and  7253/54  and  was  standardized  across  ATCFs.  Each  ATCF  provides  a  unique 
training  opportunity  for  the  Marines  stationed  there;  and  as  a  result,  Marines  training  at 
different  bases  are  exposed  to  overlapping,  but  not  quite  parallel  training  opportunities. 
For  example,  a  station  that  controls  a  complex  air  space  that  can  facilitate  a  variety  of 
aircraft  operations  may  provide  opportunity  for  Marines  to  train  on  a  range  of  real- 
world  scenarios.  Meanwhile,  Marines  at  bases  with  less  traffic  and  less  complexity  in 
airspace  are  not  exposed  to  these  challenges  as  readily  or  consistently.  This  variability 
in  exposure  is  likely  to  facilitate  training  on  some  bases,  and  allow  Marines  to  more 
quickly  qualify  on  their  positions.  Given  this  set  of  circumstances,  standardization  was 
necessary  for  time  to  quality  across  ATCFs,  such  that  those  who  quickly  qualify  at  their 
respective  stations  will  be  acknowledged  as  high  performing  trainees. 

There  was  considerable  variability  among  each  ATCF;  therefore  a  standardization 
technique  was  used  to  place  performance  scores  by  ATCF  on  a  common  scale.  For 
example,  a  high  performer  at  Cherry  Point  ATCF  may  qualify  in  four  months  whereas  a 
high  performer  at  Yuma  ATCF  may  qualify  in  six  months.  In  this  case,  standardized 
scores  per  ATCF  will  recognize  high  performers  across  various  units  and  not 
differentiate  them  when  site  factors,  not  personnel  attributes  or  ability,  account  for  the 
time  differentials. 

Qualification  test  scores  (written  exam  and  performance  exam)  were  collected  and 
averaged  for  a  GPA  for  7257,  7252,  and  7253/54.  The  GPAs  were  also  standardized  by 
ATCF.  ASVAB  (GT  Scores)  were  also  collected. 


Results 

The  resulting  data  from  the  ATC  facilities  were  cleaned  and  subjected  to  various 
analyses  including  validity  analysis.  Data  for  the  schoolhouse  is  still  being  collected  and 
so  results  are  not  yet  available.  This  section  describes  the  data  obtained  and  analyses 
with  the  results  reported  up  front.  Validity  analysis  determined: 
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•  NCAPS  scores  are  a  valid  predictor  of  success  for  Marine  Air  Traffic  Controllers 
in  the  Operating  Forces.  Success  in  the  operating  forces  is  determined  by  time  to 
7257  qualification. 

•  ASVAB  GT  Scores  did  not  correlate  significantly  with  performance  for  the 
operating  forces  as  did  NCAPS  measures. 

•  Regression  analyses  clearly  indicate  that  non-cognitive  measures,  especially  those 
assessed  with  the  Marine  Air  Traffic  Control  Suitability  Test  (MATC-ST),  add 
incremental  validity  above  the  ASVAB  GT  Score. 

Background  and  demographics 

This  section  describes  the  sample  of  participants  who  took  part  in  the  study,  as  well 
as  the  characteristics  of  the  performance  measures  collected  from  those  participants. 
Table  1.1  presents  background  and  demographic  information  for  208  Marine  Corps  ATC 
operating  force  (7257,  7252,  7253/54)  participants.  The  majority  of  the  sample  was  male 
(92.3%),  White  (63%),  with  less  than  four  years  of  service  (70.5%).  Most  of  the  sample 
was  between  Lance  Corporal  (LCpl;  E3  paygrade)  and  Sergeant  (Sgt;  E5  paygrade), 
(88%). 
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Table  1 

Operating  Forces  Demographics 


Frequency 

Percent 

Cumulative 

Percent 

Gender 

Male 

192 

92.3 

92.3 

Female 

16 

7.7 

100.0 

White 

131 

63.0 

63.0 

Hispanic 

33 

15.9 

78.9 

Ethnicity 

Black 

14 

6.7 

85.6 

Asian  or  Pacific  Islander 

10 

4.8 

90.9 

American  Indian 

3 

1.4 

92.3 

Other 

16 

7.7 

100.0 

E2  (PFC) 

3 

1.4 

1.9 

E3  (LCPL) 

44 

21.2 

23.1 

Rank 

E4  (CPL) 

81 

38.9 

62.0 

E5 (SGT) 

58 

27.9 

89.9 

E6 (SSGT) 

19 

9.1 

99.0 

E7 (GYSGT) 

2 

1.0 

100.0 

>4 

146 

70.2 

70.5 

YoS 

5-9 

53 

25.5 

96.1 

<10 

8 

3.8 

100.0 

Beaufort 

20 

9.6 

9.6 

Cherry  Point 

46 

22.1 

31.7 

Camp  Pendleton 

39 

18.8 

50.5 

Unit 

Futenma 

4 

1.9 

52.4 

1  wakuni 

4 

1.9 

54.3 

Miramar 

20 

9.6 

63.9 

New  River 

27 

13.0 

76.9 

Quantico 

28 

13.5 

90.4 

NCAPS  Data  Summary 

At  the  time  of  this  report,  complete  data  were  obtained  from  the  NCAPS  measure  for 
130  Marine  Corps  ATC  Schoolhouse  (7251 MOS)  participants  and  847  Marine  Corps 
ATC  operating  force  (7257,  7252,  7253/54)  participants. 

Of  the  19  NCAPS  components,  four  were  related  to  attributes  relevant  to  the  current 
research  effort.  .  For  each  NCAPS  component,  the  data  consist  of  a  theta  value  that 
describes  the  participants  standing  on  the  construct  being  measured,  and  a  PSD  or 
posterior  standard  deviation,  which  is  an  index  of  algorithm  convergence.  A  theta  value 
is  defined  as  an  individual’s  standing  on  a  construct  being  measured  using  adaptive 
testing,  or  item  response  theory,  where  items  are  presented  to  individuals  based  on  their 
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responses  to  prior  items.  This  process  continues  until  the  algorithm  closes  in  on  that 
individual’s  standing  on  the  construct,  which  is  represented  by  the  value  of  theta. 

The  four  relevant  NCAPS  dimensions  for  ATC  operating  forces  performance  were: 
Adaptability/ Flexibility  (+),  Vigilance  (+),  Empathy  (-),  and  Self-Reliance  (-).  The 
degree  of  relevance,  or  relatedness  (validity)  was  derived  through  regression  analyses, 
which  identified  these  four  dimensions  (or  traits)  as  statistically  significant  predictors  of 
ATC  performance  in  the  operating  forces.  (Regression  analysis  was  not  conducted  on  the 
ATC  schoolhouse  data  at  the  time  of  this  report  because,  as  mentioned  earlier, 
schoolhouse  data  collection  is  still  in  progress).  These  NCAPS  dimensions  were  used  to 
develop  the  algorithm  for  the  Marine  Air  Traffic  Control  Suitability  Test  (MATC-ST). 

Operating  Forces  Performance  Data  Summary 

The  ATC  operating  forces  performance  data  summary  in  Table  2.1  includes  the 
minimums,  maximums,  means,  modes,  and  standard  deviations  of  GT  Score,  7257  GPA, 
and  7257  performance  score  (standardized  days  to  7257  qualification).  GT  Scores  were 
collected  for  187  ATC  operating  forces  Marines,  with  a  minimum  GT  Score  of  102,  a 
maximum  GT  Score  of  142,  a  mean  GT  Score  of  115.71,  the  mode  GT  Score  was  114,  and 
the  standard  deviation  for  GT  Score  was  8.268. 

The  7257  GPA  was  collected  for  122  ATC  operating  forces  Marines,  with  a  minimum 
GPA  of  83%,  a  maximum  GPA  of  100%,  a  mean  GPA  of  93.22%,  the  mode  GPA  was  94% 
and  the  standard  deviation  for  GPA  was  4.10%.  The  performance  score  (standardized 
days  to  7257  qualification)  was  collected  for  208  Marines,  with  a  minimum  performance 
score  of  (-3.12),  a  maximum  performance  score  of  1.96,  a  mean  performance  score  of 
.0024.  The  standard  deviation  for  performance  score  was  .98727. 


Table  2 

Operating  Forces  Performance  Descriptive  Statistics 


N 

Minimum 

Maximum 

Mean 

Mode 

Std.  Deviation 

GT  Score 

187 

102 

142 

115.71 

114 

8.268 

GPA  7257 

122 

83 

100 

93.22 

94 

4.100 

Performance 

Score 

208 

-3.12 

1.96 

.0024 

.98727 

Marine  ATC  Suitability  Test  Score  and  Validity  Analysis 

The  Marine  ATC  Suitability  Test  (MATC-ST)  score  is  calculated  from  four 
dimensions  of  NCAPS  that  were  submitted  in  regression  analysis  with  the  operating 
forces  Performance  score  as  the  performance  criterion.  This  section  describes  the 
correlation  and  regression  analyses. 
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Table  2.2  presents  the  intercorrelations  of  the  predictor  test  scores,  (MATC-ST), 
ASVAB  GT,  and  the  criterion,  operating  forces  performance  scores. 


Table  3 

Predictor  Test  Scores  and  Performance  Score  Intercorrelations 


Suitability 

Score 

Performance 

Score 

GT  Score 

GPA7257 

MATC- 

Suitability  Test 

1 

Score 

Performance 

Score 

.375** 

1 

GT  Score 

-.006 

.077 

1 

GPA7257 

.030 

.110 

.064 

1 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 

Table  2.2  shows  that  the  only  correlation  that  was  statistically  significant  was 
between  the  MATC-Suitability  and  the  Performance  score,  (R  =  .375,  p<.oi).  For 
completeness,  the  data  were  used  in  a  hierarchical  regression. 

Through  hierarchical  regression,  the  prediction  of  the  MATC-ST  score  on 
performance  outcomes  may  be  isolated  and  compared  to  the  prediction  of  other  factors, 
such  as  the  ASVAB  score,  as  well  as  the  prediction  of  composites  of  multiple  factors  at 
once.  This  technique  uses  comparisons  of  successive  regression  models  and  determines 
the  significance  that  each  one  has  above  and  beyond  the  others.  Three  models  were 
tested  and  compared:  (1)  the  ASVAB  GT  scores’  prediction  of  Performance  scores,  (2) 
the  MATC-  Suitability  Test  scores’  prediction  of  performance  scores,  and  (3)  regression 
weighted  composite  scores’  (made  up  of  ASVAB  GT  and  MATC-Suitability)  of 
Performance  scores. 

The  Marines  currently  use  the  ASVAB  as  a  selection  screening  tool  to  gain  access  to 
the  ATC  School;  therefore,  GT  Score  was  the  first  variable  entered  (model  1),  followed  by 
the  MATC-Suitability  Test  score  (model  2),  and  the  resulting  composite  (model  3). 
Results  of  the  regression  analysis  are  presented  in  Table  2.3  (weights  are  standardized 
beta  (p)  weights  and  thus  represent  correlations). 
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Table  4 

ATC  Operating  Forces  Hierarchical  Regression  Validity  Analysis 


Measure 

Standardized 

Regression 

Coefficient 

t 

Probability 
of  Sig. 

GT  Score 

.082 

1.127 

.261 

MATC  Suitability 
Score 

•375 

5.804 

.000* 

GT  Score  +  MATC 

Suitability 

•371 

5-304 

.000* 

a.  Dependent  Variable:  Performance  Score; 

"Statistically  significant  at  the  p<.ooi 

probability  level. 

Table  2.3  results  showed  that,  as  expected  from  the  results  in  table  2.2,  GT  Score 
alone  (model  1)  does  not  predict  Performance  Score  at  a  statistically  significant  level 
(model  1;  p=.o82,  t  =1.127,  p<-26i).  However,  when  the  MATC-Suitability  was  added 
into  the  next  step  (Step  2)  of  the  hierarchical  regression  analysis,  the  results  were 
statistically  significant  (model  2;  P=-375,  t=5.8c>4,  p<.ooo).  The  .082  standardized 
regression  weight  for  GT  and  .375  for  MATC-Suitability  are  the  exact  correlations 
reported  in  Table  2.2,  just  confirming  in  regression  analysis  that  two  variables  do  not 
overlap  and  that  adding  MATC-Suitability  to  the  operational  GT  selection  system  adds 
unique  variance  accountability. 

As  expected  from  Table  2.2’s  statistically  non-significant  -.006  correlation  between 
MATC-Suitability  and  GT,  a  composite  of  the  two  variables  entered  in  the  third  model 
(Step  3)  did  not  improve  the  prediction  over  MATC-Suitability  alone.  (Model  3;  p=-37i, 
t=5-304,  p<.ooo).  That  is,  the  model  was  only  significant  due  to  the  fact  that  MATC 
Suitability  Test  Score  has  a  significant  predictive  relationship  to  Performance  Score 
(again,  model  2;  p=.375>  t=5.804,  p<.ooo). 

Figure  1.1  provides  a  graphical  representation  of  the  predictive  relationship  between 
GT  Score  and  Performance  Score  (R2=.oc>7).  Figure  1.2  provides  a  graphical 
representation  of  the  predictive  relationship  between  MATC  Suitability  Test  score  and 
Performance  Score  (R2=.i4i).  Finally,  Figure  1.3  applies  to  the  combined  GT  and 
MATC-Suitability  as  the  predictor.  Figure  1.3  applies  to  the  combined  GT  and  MATC- 
Suitability  as  the  predictor  (R2=.i38).  R2  (variance  overlap)  is  shown  on  each  graph  and 
is  the  proportion  of  the  Performance  Score  that  is  accounted  for  by  the  predictor.  The 
square  root  of  this  value  is  the  correlation.  As  a  note  for  each  graph,  the  performance 
score  was  reverse  coded  so  that  the  true  relationship  of  less  time  to  7257  quality  being 
associated  with  higher  predictor  scores  could  be  displayed  as  positive. 
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Performance  Score 


R2  Linear  =  0.007 


-4.00- 


~ I - 1 - 1 - 1 - 1 - 1 

100  110  120  130  140  150 

GTScore 


Note. 

l. Performance  score  is  reverse  coded. 

Figure  1.  GT  Score  and  Performance  Score 
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Performance  Score 


Note. 

l.  Performance  score  is  reverse  coded. 

Figure  2.  MATC-Suitability  Test  Score  and  Performance  Score 
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R2  Linear  =  0.138 
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~r~ 
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~i - 1 - 1 - 1 - r~ 
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GT  +  Suitability  Score 


Note. 

l. Performance  score  is  reverse  coded. 

Figure  3.  GT  Score  and  MATC-Suitability  Test  Score  Composite  and 

Performance  Score 

Together,  GT  Score  and  MATC-  Suitability  Test  Score  account  for  13.8%  of  the 
variance  in  performance  score  whereas  MATC-Suitability  alone  accounts  for  a  slightly 
greater  14.1%.  The  “variance  accounted  for”  metric  interpretation  of  a  correlation 
provides  meaningful  insights  into  the  value  of  a  selection  &  classification  instrument. 
That  is,  predictor  test  developers  strive  for  100%  performance  variability  accounted  for 
by  the  predictor  (a  correlation  of  1.0).  However,  to  set  a  cutscore  for  an  operational 
selection  standard,  the  correlation  coefficient  (validity  coefficient)  is  required.  The 
cutscore  topic  will  be  fully  explained  after  the  data  from  the  next  cohort  going  through 
the  ATC  schoolhouse  can  be  analyzed. 

At  the  time  of  this  report  attrition  outcomes  were  not  present  in  the  ATC  operating 
forces  data.  Some  additional  regression  analysis  was  completed  adding  three  NCAPS 
components:  Self-Control  (-),  Perceptiveness/Depth  of  Thought  (+),  and  Willingness  to 
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Learn  (+).  Although,  when  these  three  dimensions  were  combined  with  the  previous 
four  dimensions,  the  seven  trait  score  was  still  a  significant  predictor  of  Performance 
Score  (P=-390,  t=6.o68,  p<.ooo,  R2=.i52);  any  recommendations  using  this 
combination  will  need  a  larger  data  set  and  predictive  validity  gained  from  the  ATC 
Schoolhouse  population. 

The  MATC-ST  has  been  shown  to  be  predictive  of  success  for  Marine  ATCs  in  the 
operating  forces.  Success  is  determined  by  time  to  quality,  therefore  the  higher  the 
MATC-ST  score  the  less  time  it  took  for  an  operating  forces  ATC  Marine  to  quality  as  a 
7257.  The  expectation  is  that  the  relationship  will  hold  up  for  future  ATC  Marines  and 
that  a  near  term  goal  would  be  to  evaluate  the  positive  effects  of  an  operational  cutscore. 


Recommendations 

The  Marine  ATC  Suitability  Test  (MATC-ST)  score  is  a  valid  algorithm  derived  from 
dimensions  of  NCAPS.  At  the  time  of  this  report  the  MATC-ST  is  shown  to  be  predictive 
of  success  for  Marine  ATCs  in  the  operating  forces  in  a  concurrent  validity  design. 
Further  data  collection  and  analysis  at  the  ATC  Schoolhouse  will  provide  confirmation 
of  the  MATC-ST’s  predictive  validity,  both  for  training  performance  and  further  out  to 
7257  qualification.  It  is  best  practice  to  combine  concurrent  validity  with  predictive 
validity  from  the  ATC  Schoolhouse  in  order  to  recommend  any  conservative  cutscores 
and/or  implementation  into  the  Marine  Corps  Air  Traffic  Control  selection  and 
classification  process.  Upon  completion  of  the  ATC  Schoolhouse  data  analysis, 
adjustments  may  or  may  not  be  made  to  the  MATC-ST  equation.  Subsequently,  NPRST 
and  the  Marine  Corps  will  be  in  a  position  to  assess  an  operational  cutscores. 

If  the  MATC-ST  is  to  be  utilized  for  selection/classification  it  is  recommended  that  it 
be  used  only  in  addition  to  the  ASVAB,  and  implemented  at  the  Military  Entrance 
Processing  Station  (MEPS)  prior  to  a  Marine  ATC  MOS  designation/assignment.  This 
will  reduce  attrition  and  increase  the  quality  of  Marines  selected  for  the  ATC  MOS. 

In  addition  to  the  MATC-ST,  an  interest  measurement  tool  can  also  be  designed  to 
be  implemented  at  the  MEPS  to  give  a  more  accurate  depiction  of  the  ATC  field.  During 
this  study,  some  Marines  commented  that  as  a  recruit  they  were  told  to  select  ATC,  but 
then  later  they  could  get  the  job  they  really  wanted  in  the  Marines  if  they  failed  out  of 
school.  This  is  not  an  accurate  depiction  of  the  MOS  reassignment  process  and  leads  to  a 
disconnection  for  Marine  Corps  students.  It  is  recommended  that  a  protocol  for 
Recruiters  is  established  when  screening  recruits  for  ATC  that  includes  an  interest 
measure.  One  tool  that  the  Navy  uses  in  re-classifying  is  the  Jobs  and  Occupational 
Interests  in  the  Navy  (JOIN).  This  is  an  operational  computerized  interest  inventory 
developed  by  the  Navy  for  gauging  Sailor  interest  at  the  time  of  reclassification.  A 
similar  measure  can  be  designed  for  Marine  Corps  Air  Traffic  Controllers  for  gauging  a 
Marine’s  interest  at  the  time  of  enlistment  or  reclassification. 


21 


Limitations 


The  length  of  this  study  was  approximately  one  year.  Due  to  the  time  constraints 
there  were  some  limitations  in  data  collection  for  this  study.  Although,  NCAPS  allowed 
for  maximum  participation  among  Marine  Air  Traffic  Controllers  in  the  operating  forces 
because  it  was  web  enabled,  the  performance  data  collection  was  not  as  abundant.  It  was 
necessary  for  an  adequate  amount  of  time  to  pass  to  collect  performance  data  from  the 
ATC  operating  forces  and  ATC  Schoolhouse. 

There  were  nine  different  MATCFs  included  in  this  study  and  each  one  was  visited 
(with  the  exception  of  Futenma  and  Iwakuni  because  they  were  in  Japan).  The 
performance  measurement  records  varied  by  each  MATCF.  Time  to  qualify  as  a 
performance  criterion  was  collected  for  7257,  7252,  and  7253/54  using  start  date  and 
qualification  date.  However,  the  start  dates  for  senior  ATCs  often  did  not  coincide  with 
7257  qualification  dates,  and  there  was  not  enough  data  collected  for  7252  and  7253/54 
qualifications.  The  bulk  of  the  7257  time  to  qualification  data  came  from  the  ranks  of  E- 
6  and  below.  There  were  also  limitations  in  the  7257  GPA  quantity  of  data  collection 
from  the  operating  forces  because  not  every  MATCF  maintained  7257  qualification  test 
scores  (i.e.  Cherry  Point). 

Perhaps  the  biggest  limitation  at  the  time  of  this  report  is  that  the  ATC  Schoolhouse 
data  collection  and  analysis  is  not  complete.  NCAPS  and  performance  data  collection  at 
the  schoolhouse  is  still  in  progress.  An  additional  report  will  need  to  be  produced  at  the 
time  of  the  ATC  Schoolhouse  data  analysis  completion. 


Conclusion 

The  greatest  outcome  of  this  study  even  if  only  at  the  ATC  operating  forces  level  is 
the  answer  to  a  long  dwelled  upon  question  of:  What  individual  characteristics  will 
predict  success  in  the  Air  Traffic  Control  job?  Past  Marine  Corps  studies  were  unable  to 
link  revocations  in  ATC  to  ASVAB  (GT  Scores)  or  other  cognitive  based  measures.  This 
study  has  revolutionized  this  field  by  providing  a  non-cognitive  measurement  tool  that 
predicts  success  in  Air  Traffic  Control. 

As  with  any  test,  it  is  important  to  continue  to  validate  these  tools  considering 
changes  in  curriculum  and  additions  of  simulation  based  training  in  order  to  see  if  the 
ASVAB  composites  and  MATC-ST  cutscores  are  adequate  for  producing  a  job  ready 
Marine. 
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